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The fir (finite impulse response) filter is an essential tool for a 
large number of applications in communication. In this paper we 
consider the design of linear-phase fir digital filters with finitely 
precise coefficients. Coefficient inaccuracy is known to degrade the 
frequency response of band-select fir filters, especially in the stop- 
band region. We derive a bound on the attainable stopband atten- 
uation, and we also develop techniques for designing fir filters with 
finitely precise coefficients. Mixed-integer programming algorithms 
are presented to select finitely precise coefficients for a filter that best 
approximates an arbitrary magnitude characteristic in the minimax 
sense. Our method generates a number of possible solutions including 
that of simple rounding or truncation and then selects the best finitely 
precise coefficients from this set. In this way, significant improvement 
in the filter performance is gained over methods that simply round or 
truncate the infinitely precise coefficients. We also show how integer 
programming can be used to design filters with powers of two coeffi- 
cients. Such filters are easier to mechanize since they do not require 
multipliers. 



I. INTRODUCTION AND SUMMARY 

In this paper, we develop techniques for designing linear-phase finite 
impulse response (fir) digital filters with finitely precise coefficients. 
The fir filter has wide applications in communications. 1 For example, 
fir filters have been used as band-select filters in fdm/tdm translators 2 
and in Touch-Tone® receivers. 3 They have been used as adaptive 
equalizers, 4 as matched filters in radar, and as echo cancellers in 
satellite communications. 5 In addition, they are widely used in speech 
synthesis and analysis. 6 Many techniques are available for the design 
of infinite-precision fir filters. Such techniques include (i) the use of 
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windows, 7 (ii) the linear programming approach, 8-10 (Hi) the Remez 
exchange algorithm by Parks and McClellan, 11 (iv) the interpolation 
techniques by Hofstetter et al, 12 (v) Hamming-Kaiser twicing algo- 
rithms 13 and (vi) the nonlinear optimization procedure by Herrmann. 14 
The above techniques are appropriate for filters with infinitely precise 
coefficients, i.e., for sampled data systems. If a sufficient number of 
such coefficients are used, fir filters can be designed to approximate 
virtually any frequency response as closely as desired. 

fir filters are commonly implemented either as charge-coupled 
devices (ccds), as surface acoustic wave (saw) devices, or as digital 
filters. The first two allow the realization of fir filters without the 
need for analog-to-digital conversion. However, there are fundamental 
limitations on the attainable coefficient accuracy for ccds and saw 
devices. 15 Current ccd technology can only mechanize fir filters with 
coefficient accuracies of 10 bits. Finite-precision ccd filters have been 
designed using integer programming techniques. 16 In digital filtering,* 
fir filters, as depicted in Fig. 1, and other digital signal-processing 
algorithms are implemented either with special-purpose hardware or 
as programs in digital computers or signal processors. In either case, 
the data sequence values {x(n)} and coefficients {h(n)} are usually 
stored in finite-length registers or memory elements. Register length 
is an important economic factor in hardware implementations. Re- 
cently, innovations in hardware have emphasized the importance of 
efficiently designing digital filters with finite register length coeffi- 
cients. Of particular interests are the advances in microprocessors, 17 
bit-slice technology, 18 and programmable digital signal processors. 19,28 

In this paper, we discuss only one effect of finite register length in 
digital filters, i.e., quantization of the coefficients. Usually the coeffi- 
cients of a digital filter are obtained by some theoretical or optimization 
design procedures that essentially assume an infinitely precise repre- 
sentation of the filter coefficients. As a consequence, the frequency 
response of the quantized (or digital) filter can deviate appreciably 
from the filter designed with infinitely precise coefficients. In fact, the 
quantized filter may in certain cases fail to meet initial specifications 
even though the unquantized filter does. As an example, consider the 
low-pass filter specification shown in Fig. 2. The desired filter should 
have a stopband attenuation of 80 dB, a ripple of 0.01 dB in the 
passband, and a transition ratio of 0.6. An optimum filter of length 49 
designed with infinite precision coefficients using linear programming 
techniques had a stopband attenuation of 98.6 dB and a passband 
ripple of 0.002dB (see curve A in Fig. 3). However, rounding the filter 
coefficients to 12 bits resulted in a minimum stopband attenuation of 



* The coefficient precision of the fir filter can be made as accurately as required at 
the price of increasing hardware cost. 
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Fig. 1 — fir filter. 



62 dB. This indicated a degradation in the stopband of approximately 
32 dB. This result is shown as curve B in Fig. 3. We see from this 
example that coefficient inaccuracy can degrade the frequency re- 
sponse of band-select filters, especially in the band-reject region. The 
number of bits would have to be greater than 12 bits to meet the 
original specification. 

A useful way of finding the appropriate number of bits is to use 
formulas that estimate or bound 20-22 the error magnitude in the fre- 
quency domain. These bounds are functions of the number of bits and 
length of the filter. In Section II we derive an upper bound on the 
error magnitude in the frequency domain. For the filter example given 
in Fig. 2, known existing bounds and our new bound indicate that 14 
to 17 bits of coefficient wordlength would be required to achieve a 
stopband attenuation of 80 dB. These bounds only serve as a guide for 
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Fig. 2 — lpf specification. 
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Fig. 3 — Spectra of filter with infinitely precise and rounded coefficients. 

the design of band-select filters. In an actual implementation, the 
frequency response is evaluated for the designed filter using different 
coefficient wordlengths to determine the least number of bits needed 
to meet the desired specification. The coefficient wordlengths used in 
a trial are in the proximity of the wordlength determined by the 
bounds. Though these bounds only serve as guides for the design of 
fixed fir filters, they are extremely useful in establishing the coefficient 
wordlength requirements for adaptive fir filters, 24 because for adaptive 
filters it is not possible to know a priori what the exact desired 
characteristics are. 

For the filter example discussed earlier, an improvement of 2 dB in 
the stopband attenuation is achieved by using selected rounding (i.e., 
a mixture of rounding, truncation, or boosting) rather than rounding 
off the coefficients to 12 bits. Other quantization schemes such as pure 
truncation in either two's complement or sign-magnitude format sim- 
ilarly degraded filter performance (see Table I for comparison of the 
various rounding schemes). We display in Table II the filter coefficients 
for the infinitely precise filter, rounded filter, truncated filter, and a 
random-rounded filter where we show only half of the coefficients 
starting from the center coefficient. In Table III we find that truncation 
can produce a better result than rounding. 

We develop, in Section III, techniques that include the effects of 
coefficient quantization in the design of fir filters. We use mixed- 
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Table 1 — Rounded and truncated stopband ripples 
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integer linear programming algorithms to select finitely precise coef- 
ficients for a fir filter that approximates a given magnitude charac- 
teristic. These mixed integer routines allow the unknown coefficients 
to take on integer values while the stopband and passband ripples are 
allowed to be noninteger. The approximation is optimal in the given 
design sense, i.e., minimum absolute-weighted value of the error in the 
filter response. The errors in the approximation cannot be both equi- 
ripple and minimax, as in the case of infinite precision designs using 
linear programming, because of the finite precision wordlength con- 
straints. However, other desired properties of fir filters such as linear 
phase can still be preserved by constraining the resultant finitely 
precise coefficients to be even and symmetric. 

The two mixed-integer programming optimization methods we use 
are the well-known branch and bound algorithm 23 and the zero-one 
algorithm 26 using decomposition methods. We introduce several con- 
Table II — Coefficient values 

(Length of Filter = 49, Sampling Frequency = 1.0, 
Passband Frequency = 0.16875, Stopband Frequency = 0.28125) 
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Table III — Stopband ripples for rounding and 
truncation 

(Length of Filter = 33, Sampling Frequency = 1.0, 
Passband Frequency = 0.15, Stopband Frequency = 0.3, 

Infinitely Precise Coefficients Stopband Ripple = -79.12 dB) 
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straints to speed the convergence time of the optimization. These 
constraint techniques allow the designer to round the infinitely precise 
coefficients to the nearest M variable neighborhood, where M is the 
number of LSB bits which were allowed to vary. We especially con- 
centrate on the simplest case of M = 1, i.e., the unit neighborhood. In 
addition, the constraints allow the designer the flexibility of fixing 
some coefficients while varying others. Although the use of our con- 
straints reduced the convergence time of the optimization algorithm, 
the approximation error increased. The error (i.e., the deviation be- 
tween the magnitude response of the ideal and quantized filters) in the 
approximation using either the unconstrained or constrained mixed- 
integer programming techniques is compared to the error introduced 
by either straight rounding or truncation of the coefficients. In some 
design examples, we obtained improvements of 7 dB in the stopband 
attenuation between the amplitude response of the optimized filter 
with finitely precise coefficients and that of the filter obtained by 
rounding the infinitely precise coefficients. 

Section IV shows how to use zero-one integer formulation to design 
fir filters. We compare the results of a zero-one integer design with 
the mixed-integer design discussed in Section III. Finally, we show 
how to design filters with powers of two coefficients. 

II. THE TRANSFER FUNCTION 

The transfer function of a linear-phase fir filter of length N has the 
general form 

(N-D/2 

H a (z)= £ h n z~ n , (1) 

n=-(N-l)/2 

where the length N is odd and h n = h- n . The coefficients h n are real 
and can take on any value. The frequency response, H a {e j2nf ) is 
obtained by evaluating H a (z) along the unit circle, z = e j2 " f , 

(N-D/2 

fio + 2 X h k cos(27rfk) , (2) 

*=i 



H a (e J2 " f ) = e - j2 ^ N - l)/2 - 
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where /is the normalized frequency variable and (N — l)/2 is the delay 
of the filter. The magnitude function, H(e J2vf ), is given by 

(N-D/2 

H(e J2vf ) = ho + 2 £ h k cos(27rfk), for 2V odd (3) 

N/2 

H{e j2nf ) = 2 £ h k cos(2>rrfk), for 2V even. (4) 

The distinction between N odd and N even is of considerable impor- 
tance in the design and mechanization of fir filters. 27 However, for the 
sake of simplicity, we assume hereafter that the length N is odd. We 
use the symbol h'k to represent the infinitely precise coefficients. 

In an actual implementation, the infinitely precise coefficients {h'k) 
are quantized to take on values {hk} which are integer multiples of the 
smallest quantizing step size, 2" fl , where B is the number of bits used 
for the implementation. The difference between the quantized and 
infinitely precise coefficient is defined by 

Shk = h'k — hk. (5) 

If the quantizing scheme is rounding, then 

\8h k \< 1/2- 2~ B , (6) 

and, for truncation, 

|M*|<2" B . (7) 

If the relationship between changes in the desired frequency response 
H(e j2 * f ) and the coefficients {h'k} are known, the degradation in 
performance of the quantized filter can be bounded. Such a relationship 
has the general form: 



8H(e j2 " f ) 8 
8hk 8hk 



(N-D/2 

h'o + 2 2 h'kCOs(2iTfk) 



(8) 



= 1 if& = (9) 

= 2cos{2irfk) if k *0. (10) 

Using eq. (8), we obtain the frequency sensitivity function, &H(e j2 " f ), 
which is expressed in the following form: 

(jv-n/2 

&H(e J2irf ) = 8h' Q + 2 X 8h'kCos(2iTfk). (11) 

k=i 

Assuming the largest change occurs in each coefficient, a worst case 
bound on AH(e j2,rf ) is given by 

(N-D/2 

1 + 2 X |cos 2irfk 



j2irf\ .-- n-(B + \) 



MHe Jlv ') < 2 



*=i 



(12) 
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The above equation is the Chan and Rabiner deterministic bound. 20 
We can improve this bound if we regard Sh'k as a variable which can 
take on a maximum value of 2 ~ {B+1) rather than assume it is a constant. 
Then a bound on the frequency sensitivity function is obtained in the 
appendix as 



AH(e' 2 *0 < 2-< B+1) . y^N' + j- \-W N (f) 



(13) 



where 



(N- 1) + [sin(iV27r/-)/sin(27r/)] 1/2 

w N (f) ^n=T) • (14) 

The weighting function, W N ( f), takes on a value of 1 at / = and / = 
%, i.e., 

W N (0) = Wn^A) = 1 

and 

Wn(0 ^1 for all other values of/". 

Therefore, a frequency-independent upper bound on &H(e i2 " f ) is given 
by 



MI{e^ f ) < 2- (fl+1) . yj N 2 +j-\- (15) 

In the limit as N — > oo 

MmWW/)-l f 0</<i 

AT^oo V2 2 

and an approximate bound on the frequency sensitivity function for 
large N becomes 

<4=-2- (B+1) .iV. (16) 

V2 

We refer to the above bound as the L-2 norm bound, and, in comparison 
to the Chan-Rabiner deterministic bound, 20 the L-2 norm bound is 
tighter by a factor of 1/V2, or 3 dB. However, this deterministic bound 
is not as tight as the statistical bounds of Gersho et al 21 and Chan- 
Rabiner. 20 The basic assumptions of the statistical bounds is that the 
quantizing error in each coefficient is randomly distributed. In Fig. 4, 
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Fig. 4 — Bounds on frequency sensitivity function. 

we compare the stopband attenuation derived from these bounds, for 
a filter of length 33 with passband and stopband normalized frequencies 
of 0.15 and 0.3, respectively. We also display in Table IV the appro- 
priate number of bits required for such a filter to have a stopband 
attenuation of at least 45 dB. 

In Table IV, we find by using our technique that an optimized filter 
would require only 8 bits, while the bounds suggest wordlengths of 10 
to 13 bits. This indicates that significant improvements can be achieved 
in filter performance if the filter coefficients are optimized; that is, by 
including the effects of quantization in the direct design of fir filters. 
In the next section we present various techniques for designing fir 
filters with finitely precise coefficients. 



III. FORMULATION OF THE DESIGN PROBLEM 

We now address the problem of finding an optimum set of coeffi- 
cients that give the best approximation of H(e j2v ')) to a desired 
magnitude function D(e i2vf )) in the minimax sense. We denote the 
error in the approximation by 

E(e J2vf ) = D{e j2 ' f ) - H(e i2 ' f ), (17) 
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where < /"< x h. The desired magnitude function D(e j2nf )) is a real- 
valued function which for our purposes will be defined only at discrete 
set of frequencies {/*} where k = 1, 2, •••, K. The choice of the 
discrete set of points {/*} is of considerable importance. Equation (17) 
can be written in vector form as: 

E = D - H. (18) 

Rabiner showed that an optimal set of infinitely precise coefficients 
{hi} that best approximates H to D in the minimax sense is obtained 
by formulating the problem as a linear programming problem. 10 

3. 1 Linear programming formulation 

For simplicity of exposition, the linear programming problem is 
formulated for a low-pass filter that best approximates the desired 
magnitude characteristics shown in Fig. 2. The passband and stopband 
frequencies are f p and f s , respectively. The desired magnitude function 
D(e J2,rf ) is represented mathematically as 

D{e j2 " f ) = l forO<A</ P 

= for/ 8 </-*<Vfe. (19) 

From Fig. 2, the constraints on the low-pass filter are: 

1 - 5i < H(e J2nf ) < 1 + Si for < /* < f p passband 

H(e J2 " f ) < 5 2 for f 8 < f k < % stopband. (20) 

The linear programming problem is to minimize 

f(8i,8 2 ) = C'X (21) 

subject to 

AX < Do (22) 

L < X < U, (23) 

where 

C' = [c u c 2 ,0,0 ... 0] (24) 

is the transpose of an [(N - l)/2] + 3 vector. The coefficients Ci and 

Table IV — Number of bits required to 

achieve a stopband attenuation of 45 dB 

for the filter specified in Fig. 2 

Chan-Rabiner Deterministic Bound 13 bits 

L-2 Norm 12 bits 

Chan-Rabiner Statistical Bound 10 bits 

Gersho, Gopinath, and Odlyzko 10 bits 

Actual Rounded Filter 10 bits 

Empirical 6-dB Estimate 9 bits 

Optimized Filter 8 bits 

1584 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1 980 



C2 are the weights of 5i and S2, the passband and stopband ripples, 
respectively. 



X = 



5i 
82 
h\ 
h' 2 

h'i 



h\ 



(N-D/2 



(25) 



is an [(N - l)/2] + 3 vector whose third to [(N - l)/2] + 3 elements 
are the filter coefficients. The first and second elements are the 
passband and stopband ripples. All the elements of X are real numbers 
with infinite precision. The matrix A is a 2K by [[(N - l)/2] + 3] 
constraint matrix whose elements are either —1, 0, or ± {2 cos 2wfki), 
where i runs from 1 to (N — l)/2. Do is a 2K vector whose elements are 
either 1, 0, or — 1. U and L are ([(N — l)/2] + 3) vectors that specify 
the upper and lower bounds on the vector X. For a low-pass filter 
whose passband magnitude is normalized to unity, the magnitude of 
the elements of U and L are all less than unity. 

3.2 Formulation of mixed integer linear programming problem 

The decision variables, h ,', obtained from the linear programming 
formulation above are the infinitely precise filter coefficients. When 
these variables are quantized to a fixed number of bits, the resulting 
solution is no longer optimal. To obtain an optimal solution, the effects 
of coefficient quantization should be included in the formulation of the 
problem. This is done by formulating the problem as a mixed-integer 
linear programming problem. In formulating a mixed-integer program- 
ming problem, the symbol B represents the number of bits, including 
the sign bit. We also scale the vectors Do, f/and L by 2 fl_1 . Substituting 
the scaled vectors into (22) and (23) results in the following mixed- 
integer linear programming formulation shown in summation notation: 



minimize C\8\ + C282 



(26) 



subject to 



N-l 
2 



-1.6*i -O.S2 ± ho ±2 X ^.cos 2irfji < ±2 



-B+l 



< /", < fp (i.e., passband) 

O.Si -I.62 ± ho ±2 X hiCOB 2-nfji < 

f s < fj < fi/2 (i.e., stopband) 



(27) 
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ai <«,<£,, (28) 

a 2 <5 2 </?2, (29) 

N- 1 
/, < hi <u it i = — . (30) 

The decision variables 5i and 8 2 are real variables bounded by ai, fii, 
oi2, ($2- The rest of the decision variables h'ts are integer variables 
bounded by integer constants less than or equal to 2 B ~ 1 . The problem 
as formulated in (26) through (30) is a mixed-integer linear program- 
ming problem. Commercial software packages are available for solving 
mixed-integer linear programming problems. 23 A particular software 
package we found useful was written by Kochman. 25 This program's 
interfaces are straightforward and can easily be integrated with other 
software packages to provide a complete system that is efficient in 
computation time and storage requirements. The additional software 
needed for efficient computation time and storage requirements are 
discussed later. 

The integer variables that result from the mixed-integer linear 
programming problem defined above are the coefficients of a linear 
phase fir filter. These coefficients are integers which he in the range 
— 2 B_1 to 2 B_1 . The actual binary bits used in the mechanization of the 
filter are the binary representation of the integer variables. We plot, in 
Fig. 5, the stopband attenuation for a filter of length 33 with passband 
and stopband frequencies of 0.15 and 0.3, respectively, using 12, 10, 8, 
6, and 4 bits. These results were obtained using mixed-integer program- 
ming design. Also plotted in Fig. 5 are the stopband attenuation 
derived using the Gersho et al. bound, 21 * the rounded solution, and 
the 6 dB/bit heuristic estimate. For all the number of bits considered, 
the optimized filter increased the stopband attenuation by 5-7 dB. 
Improvements were also observed in the passband ripples. The differ- 
ence in stopband attenuation between the optimized filter and the 
Gersho et al. bound for quantized coefficients is approximately 10 dB. 
These results clearly show the advantage of the optimized filter com- 
pared with direct quantization of the infinite-precision filter. 

3.3 Computational efficiency 

A mixed-integer linear programming problem is solved in two stages. 
First, the problem is optimized by considering all the integer variables 
as being continuous variables. The optimal solution obtained is called 
an optimal continuous solution. These are the infinitely precise filter 
coefficients {h'i). Second, a search is started from the optimal contin- 

* We used the Gersho et al. bound because in our experience it is the tightest bound. 
The Gersho et al. bound predicts performance for quantization of conventional filter 
designs and not for the mixed-integer optimized filter. 
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Fig. 5 — Comparison of bounds, rounded and optimized solution. 

uous solution obtained at the end of the first stage. The integer 
variables are then forced to take integer values using a "branch and 
bound" technique with heuristic rules. An optimal integer solution 
satisfying all the constraints and giving the best possible value to the 
objective function is searched for. Using an ordinary commercial 
mixed-integer programming software package on a filter design prob- 
lem results in long computation times,* unless several improvements 
are added to the software package. The reasons for this are twofold. 
First, the constraint matrix A is unusually large and dense.f Second, 
simple commercial software packages employ simple, straightforward 
strategies. A straightforward strategy leads to a series of integer 
solutions tending toward the optimal integer solution. When an integer 
solution is found, it is not immediately known whether it is optimal. 
The search continues either until a better solution is found or until all 
the set of possible solutions is exhausted. For problems with many 
integer variables and relatively loose constraints, good solutions are 



* This severely limits the length of filters than can be designed. 

f The constraint matrix A is very sparse for problems found in the fields of investment, 
capital budgeting, and production planning. Most commercially available, mixed-integer 
programming packages were designed for these types of problems. 
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quickly found, but a long computation is necessary either to improve 
them slightly or to prove their optimality. 

To alleviate the above problems, we introduced several improve- 
ments in our software package to enhance the computational efficiency 
of the algorithm. For example, the designer can interrupt the search 
after an integer solution has been found and change the bounds on the 
passband and stopband ripples. This decreases the computation time 
and storage requirements. Another important consideration is the 
order in which integer variables are processed during the search for 
integer solutions. This order is called "priority order" of integer vari- 
ables. Solution times vary significantly with the priority order chosen. 
Integer variables should be processed according to their importance in 
the model, the most important ones being processed first. Usually the 
integer variables are processed in the order* in which they appear in 
the decision vector X. We found in most filter design examples that 
the smaller coefficients are the most sensitive. Slight changes in their 
least significant bits produced correspondingly larger changes in the 
stopband ripple. In a particular example, (see Table II), a change in 
the 23rd coefficient for the rounded filter from one to zero produced a 
5-dB change in the stopband ripple. A change from one to zero 
represents just a change in the least significant bit. The 23rd coefficient 
is shown in a rectangular box in Table II. Therefore, eqs. (26) through 
(30) are rearranged so that the smaller coefficients appear in the 
leading rows of the decision vector X. Another facility in the software 
package is the option to freeze some integer variables. This fixes the 
activities of the integer variables to their current integer values in the 
matrix, thus allowing for post-optimal studies of the solution. This 
facility is essential for very long length filters, in particular, filters of 
length greater than 63. The computation time required to obtain either 
an optimal or a good solution increases with the length of the filter. 
For very long length filters, a good solution is obtained by dividing the 
original integer programming problem into suboptimal integer prob- 
lems using the following algorithm. 

Step 1: (Initialization) Divide the integer coefficients into two sets, 
Si and S2. Fix the coefficients in Si at their rounded values. 
Vary the coefficients in S2 for a suboptimal integer solution. 
Go to Step 2. 

Step 2: Fix the coefficients in S2 obtained from the suboptimal 
integer solution. Vary the coefficients in Si for a new sub- 
optimal solution. Go to Step 3. 

Step 3: Fix the coefficients in Si obtained from the current subop- 



* This is the case except when other priority capabilities have been included in the 
software package. 
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timal integer solution in Step 2. Vary the integer coefficients 
in S2 for a new suboptimal solution. Go to Step 4. 
Step 4: Test whether the current suboptimal solution is desirable. 

If not, go to Step 2, or else terminate. 
Such a heuristic procedure eventually produces a fairly good integer 
solution. This solution is only a local optimum. 

3.4 M variable neighborhood 

For the mixed-integer linear programming problem formulated in 
(26) through (30), the decision variables take on values ranging from 
— 2~ B+1 to 2~ B+l . Such a large range represents loose constraints which 
generally requires long computation times for filter optimization prob- 
lems. The computation time can be shortened by tightening the 
constraints or bounds on the decision variables. This restricts the 
decision variables within the neighborhood of the global continuous 
solution. This concept is referred to as the M-variable neighborhood. 
The integer solution obtained using this technique is a local optimum. 
The elements of U m and L m define the M-variable neighborhood: 

U m = •! h'i + M, i = 0, • • • ^lil , (31) 



h m = \h\-M,i = y >>>¥^±\, (32) 



where hi, is the ith infinitely precise filter coefficient obtained from 
the first part of the optimization (h'i is the scaled coefficient). Thus, 
the decision variables {hi) can only take on values that are M units or 
less from the infinitely precise coefficient h \ . The special case when M 
= 1 is referred to as the unit neighborhood. The local optimal solution 
produced by the unit neighborhood is equivalent to the best quantiz- 
ing* scheme for the continuous solution. We plot in Figure 6 the 
stopband attenuation for a filter designed using the unit neighborhood 
technique. This curve is labeled in Fig. 6 as best rounding. The length 
of the filter used was 33 and the passband and stopband frequencies 
were 0.15 and 0.3, respectively. In these cases, 12, 10, 8, 6, and 4 bits 
were used. Also plotted in Fig. 6 are the stopband attenuation curves 
for the rounded filter and the global optimized filter, f We find in Fig. 
6 an improvement of approximately 3 dB in stopband attenuation for 
the best rounded filter compared to the stopband attenuation of the 
simple rounded filter. The optimized filter has at least 5 to 7 dB 



* The quantizing scheme used in this context includes a mixture of rounding, trun- 
cation or boosting, where boosting is the opposite of truncation. In using this scheme, 
each coefficient is rounded up or down in the optimal way. 

| For the global optimized filter, M = 2 B_1 . 
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Fig. 6 — Stopband attenuation for rounded, best rounded and optimized filters. 



improvement in stopband attenuation compared to the simple rounded 
filter. Improvements were also observed in the passband ripples for 
the optimized and best rounded filters. Table V displays the filter 
coefficients for the best rounded, simple rounded, and optimized filters. 

One can select any neighborhood; the larger the value of M, the 
closer the local optimal solution approaches the global optimal solu- 
tion. A global optimum is also a local optimum with respect to any 
neighborhood containing the global optimum. Thus, enumeration of 
all local optima, with respect to all neighborhoods, may produce an 
acceptable solution. Sometimes iterations over a few neighborhoods 
would be sufficient to obtain a good solution. The neighborhood 
technique reduces the computation time, search space and storage 
requirements. . 

The unit neighborhood technique is extremely useful for very long 
length filters. These filters are extremely difficult to design using 
integer programming in a reasonable amount of computer time. For 
example, a 12-bit filter of length 63 was designed using the unit 
neighborhood technique. An improvement of 3 dB was observed in the 
stopband attenuation compared to the simple rounded filter. Improve- 
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merits were also observed in the passband. These results are displayed 
in Table VI. 



IV. ZERO-ONE FORMULATION 

Zero-one integer programming 26 is a special case of the integer linear 
programming problem formulated in (26) through (30). The decision 
variables for zero-one programming are restricted to two values, or 
1. In our experience, zero-one integer programming packages exhibit 
faster run time for digital filter design applications. The decision 
variables are the actual binary bits of the filter coefficients. The 
decision variables in eqs. (26) through (30) can be either positive or 
negative integers. To formulate a zero-one problem, the decision 
variables are transformed to take on values in the range — > 2 B , thus 
eliminating the sign variable. The new variables are: 

hi = hj + 2*-\ (33) 

hi is expressed in terms of binary values as: 

B-l 



hi = I h iq 2«. (34) 

9-0 

Substituting eqs. (33) and (34) into eqs. (26) through (30), we have 

minimize f($i, 82) = C181 + C2&2 (35) 

subject to 

b-\ \ 

B-l 



-Si ± I /io 9 2* ± 2 I £ 5,7 cos 2irfji < ± 2 B ~ l ± g(fj) 

0</,</ P (36) 

B-\ ( B-l \ 

-S 2 ± 2 ho g 2" ± 2 S ( 2 fc*2« cos 2-nfji < ± g(fj), (37) 

f s <fj<f m , a„<5„</3„, (38) 

where n = 1, or 2 and a„ and /?„ are the bounds on the stopband and 
passband ripples. 

h iq = or 1. (39) 

In eqs. (36) and (37), g(fj) is given by 

'JV 



sm, 



^-icplirt) ^ 1 0<J<K (40) 
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Table V — Coefficients for rounded, best rounded, and 
optimized filters 

(Length of Filter = 33, Sampling Frequency = 1.0, 
Passband Frequency = 0.15, Stopband Frequency = 0.3) 



Infinite 




Best 




Precision 


Rounding 


Rounding 


Optimized 


117.03603 


117 


118 


105 


79.50871 


80 


80 


77 


9.84019 


10 


9 


20 


-22.41649 


-22 


-23 


-17 


-8.16686 


-8 


-8 


-15 


9.50543 


10 


10 


3 


5.94591 


6 


6 


9 


-3.88934 


-4 


-4 


2 


-3.75361 


-4 


-4 


-4 


1.31050 


1 


1 


-2 


2.01544 


2 


2 


1 


-0.28597 








1 


-0.89044 


-1 


-1 





-0.00239 











0.30452 











0.02607 











-0.07076 












and 

g(fj = 0)=N+2. (41) 

By using relationship (40) and (41), we ensure that the coefficients are 
positive. Equations (35) through (39) are solved using a zero-one, 
mixed-integer, linear programming software package. 
Equations (35) to (39) are written in matrix form as: 

min C'X (42) 

subject to: 

A Z X < Do , (43) 

< X < 1. (44) 

The constraint matrix A z is very large and dense. Furthermore, it has 
no structure. For a 12-bit filter of length 49, the number of columns of 
A z is (25 X 12 + 2). Allowing for at least one degree of freedom for 
each coefficient, there must be twice as many rows (frequency points) 
as columns. Hence, for this example the A z matrix is a 604-by-302 
matrix. Inverting this large dense matrix several times, as is often done 
in integer linear programming, is costly and very sensitive to numerical 
errors. To alleviate this difficulty, we implemented the following 
scheme. First we solved the problem as a linear programming problem 
and then rounded each coefficient to its nearest binary value. We 
partitioned the vector X into two parts. One part of X contained the 
least (B — q) bits. The idea was to fix the q most significant bits and 
allow only the least B — q bits to vary. Second, we separated the 

1 592 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1 980 



matrix A 2 into A q and A Bq accordingly, so that the resulting integer 
minimization problem is: 

min C'X (45) 

subject to 

A Bq X Bq < (Do - A q X q ). (46) 

The new constraint matrix Aa q is smaller. For the example discussed 
earlier, by varying only the last five bits, the constraint matrix Ab p 
becomes a 254-by-127 matrix. The constraint matrix A Bg can be 
reduced further in size since the vector Xa g need not contain all the 
(B - q) least significant bits of each coefficient, hi. The larger coeffi- 
cients may be (B — q) bits while the smaller end coefficients have only 
one or two bits contained in the modified X Bq matrix. 

4. 1 Filters with powers of two coefficients 

Usually filters with powers of two coefficients can be mechanized 
easily in hardware by simple shift operations, since the binary repre- 
sentation of such coefficients has only one nonzero bit. The filter 
coefficients are constrained to be zero or powers of two by adding the 
following constraint to (36) through (39): 

I h iq = 1 (47) 

for all i = 0«-«(N-l)/2. fir linear phase filter with powers of two 
coefficients can be designed using eqs. (35) through (39) and eq. (47), 
together with a zero-one integer optimization software package. 

V. SUMMARY OF COMPUTATIONAL RESULTS 

To illustrate the effectiveness of integer optimization, we considered 
a 33-tap low-pass filter. The passband and stopband frequencies were 
0.15 and 0.30, respectively, and the normalized sampling frequency was 
1.0. The filter with infinitely precise coefficients designed using linear 

Table VI — Stopband and passband ripples 

(Length of Filter = 63, Sampling Frequency = 1.0, 

Passband Frequency = 0.1875, Stopband Frequency = 0.2625, 

Infinitely Precise Coefficients Stopband Ripple = -79.17 dB, 

Infinitely Precise Coefficients Passband Ripple = 0.001 dB) 

Best 
Rounding Truncation Rounding 

No. of bits (dB) (dB) (dB) 



Stopband Ripple 
12 -56.9 -54.4 -60.4 

Passband Ripple 
12 0.01 0.01 0.004 
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Table VII — Stopband ripples 

(Length of Filter = 33, Sampling Frequency = 1.0, 

Passband Frequency = 0.15, Stopband Frequency = 0.3, 

Infinitely Precise Coefficients Stopband Ripple = -79.12 dB) 









Optimized 












Integer 


Best 


Varying 5 




Rounding 


Truncation 


Solution 


Rounding 


Last bits 


No. of bits 


(dB) 


(dB) 


(dB) 


(dB) 


(dB) 


12 


-62.4 


-62.7 


-66.2 


-65.3 


-66.2 


10 


-49.4 


-47.0 


-55.9 


-54.4 


-55.9 


8 


-38.5 


-38.9 


-47.2 


-42.1 


-47.2 


6 


-26.5 


-29.6 


-33.8 


-29.6 


-33.8 


4 


-14.5 


-14.8 


-23.4 


-16.9 


-23.4 



programming had a stopband attenuation of —79 dB, the passband 
ripple was 0.001 dB, and the coefficients were represented in sign- 
magnitude format. We studied the effect of rounding or truncating the 
coefficients to either 12, 10, 8, 6, or 4 bits. The results for the stopband 
and passband ripples are shown in columns 2 and 3 of Tables VII and 
VIII, respectively. For the 8- and 6-bit filters, truncating the coeffi- 
cients was better than rounding. 

Optimized filters were designed using the mixed-integer technique 
for filter wordlengths of 12, 10, 8, 6, or 4 bits. In each case, we had 
improvements in the stopband attenuation of between 5 and 7 dB. 
Improvements were observed in the passband. These results are shown 
in column 4 of Tables VII and VIII for the stopband and passbands, 
respectively. We also designed filters using the unit neighborhood 
scheme (i.e., best rounding) for the different filter wordlengths. Com- 
pared to the roundoff filters we found improvements of 3 to 5 dB in 
the stopband performance. The zero-one mixed-integer formulation 
was used for designing several filters. We used the partitioning-of- 
variables technique discussed earlier to reduce computation and stor- 
age requirements. Here we found that varying the last two binary bits 
produce negligible improvement over roundoff solutions. However, we 
found that varying the last five bits of either the 12-, 10-, 8-, or 6-bit 
filters in the zero-one integer programming design produced the same 

Table VIM — Passband ripples 

(Length of Filter = 33, Sampling Frequency = 1.0, 

Passband Frequency = 0.15, Stopband Frequency = 0.3, 

Infinitely Precise Coefficients Passband Ripple = 0.001 dB) 









Optimized 












Integer 


Best 


Varying 5 




Rounding 


Truncation 


Solution 


Rounding 


Last bits 


No. of bits 


(dB) 


(dB) 


(dB) 


(dB) 


(dB) 


12 


0.005 


0.005 


0.003 


0.004 


0.003 


10 


0.02 


0.03 


0.01 


0.02 


0.01 


8 


0.06 


0.12 


0.06 


0.06 


0.06 


6 


0.36 


0.33 


0.14 


0.17 


0.14 


4 


0.39 


0.90 


0.25 


0.30 


0.25 
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Table IX — Computation time 

Infinite precision solution 1.8 seconds 

Best rounding 9.6 seconds 

Optimized solution 790 seconds 

Proof of optimality 1560 seconds 

Zero-one optimization 370 seconds 
varying 5 bits 



stopband attenuation and filter coefficients as the optimized filter 
discussed earlier. 

5. 1 Computation time 

We display in Table IX the cpu time used for the design of a 12-bit 
fir filter of length 33. The normalized passband and stopband fre- 
quencies were 0.15 and 0.3, respectively. The stopband and passband 
frequencies are displayed in Tables VII and VIII, respectively. The 
filter was designed on the IBM 370 using mixed-integer optimization 
techniques. The CPU time used to obtain the continuous solution (i.e., 
the infinitely precise coefficients) was 1.8 seconds. It took 9.6 seconds 
to obtain the best rounded filter and 790 seconds to obtain optimized 
integer filter coefficients. It took 1560 seconds, twice as long, to exhaust 
all the possible solutions. These long computation times were 
shortened using the zero-one mixed-integer optimization technique 
and varying only the least five significant bits of the coefficients. The 
cpu time used was 370 seconds. 

VI. CONCLUSIONS 

New techniques for designing minimax linear-phase fir filters with 
finitely precise coefficients have been presented. These techniques 
generate a number of possible solutions, including that of simple 
rounding or truncation, and select the best finitely precise coefficients 
from this set. In this way, significant improvement in filter performance 
is gained over methods that simply round off or truncate the infinitely 
precise coefficients. In all design examples considered, our techniques 
increased the stopband attenuation by at least 7 dB, as well as reduced 
the passband ripple, compared to techniques that simply round off the 
infinitely precise coefficients. 

It is difficult to use integer optimization to design long filters, since 
computation time as well as storage requirements are excessive unless 
specialized techniques are employed. The computation time and stor- 
age requirements were considerably reduced by using zero-one integer 
optimization with constraints on the binary bits. This technique is 
recommended for designing optimum fir filters with limited-precision 
coefficients. A simplified version of our method chooses the best 
rounding scheme for quantizing infinitely precise coefficients to a fixed 
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number of bits. The design of a 63-tap filter using the simplified 
scheme improved the stopband attenuation by 3 dB. 
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APPENDIX 

Derivation of Frequency Sensitivity Function 

To prove the upper bound on the frequency sensitivity function 
given by (13), we regard 8hh as a variable which can take on a maximum 
of 2 _(B+1) . Therefore, the maximum change in the frequency sensitivity 
response, AH(e J2 " f ) is given by: 



AHfe^) = max 



= max 

|Sty|<2-< B+1 > 



V dH(e^) 

*—MT'* hk 



(N-D/2 

8h' + £ 28h'k(coa2irfk) 



(48) 



= max 

|fi^|s2-< B+1 > 



8h'o, 8h'i, . . . , oh' 
1, 2 cos 2tt/, . . . , 2 cos 2irf 



N-l 
2~~ 

N-l 



(49) 



The above equation is the magnitude of an inner product which we 
denote by 



&H(e i2 * f ) = max • I ((Sh\ F(f))) I 

l«/i / ;|£2-(» +| > 

((8h',F(f)))\. 



max 



||8A'||s%/(JV+l)/2. 2-" ,+l 

Using Schwartz's inequaUty, the above expression simplifies to 



when 



A Jf (c-^0 < V(iV + l)/2 • 2 " (B+1) • || F( f ) || , 

Fit) 



5h'= V(N+ l)/2.2- (B+,) ■ 



\F(f)f 



(50) 
(51) 

(52) 
(53) 



Writing eq. (52) in terms of cosines we obtain 
AH(e J2vf ) < v / (Ar+l)/2.2 -,fl+1) - 



(AT-D/2 1 ' '■'■ 

1 + 4 £ COS 2 (2<7Tfk) 
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AH(e^) < V(2V+l)/2-2 



-(B+l) 



(JV-D/2 

1 + 2 £ (1 + cos A-nfk) 



1/2 



. (54) 



Using the fact that 



1 <A/ ^ ,/2 ,, sin Mr/" 
-+ X cos 2^= — '-, 

2 A-i 2sin7r/ 



MKe*" f )<J(N+l)/2.2 



-(B+D 



(2V - 1) + 



sin N27rf 
sin 2itf 



1/2 



(55) 



(56) 



< 2- ,B+1) -V(2iV-l)(N+l)/2 



(2V - 1) sin N2irf/sm 2irf 



ql/2 



(22V - 1) 



(22V - 1) 



(57) 



° ' \ i N 2 + j-l-Wn(f), 



(58) 



where 



W N (f) = 



(2V - 1) sin N2<rrf /sin 2m{ 



(2N - 1) (22V - 1) 



1/2 
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